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Establishing  the  Reliability  and  Validity  of  the  Perceptions  of 
Flight  Operations  Quality  Assurance  Questionnaire 


The  Flight  Safety  Foundation  (FSF)  defines  Flight 
Operations  Quality  Assurance  (FOQA)  as  a  program  “for 
obtaining  and  analyzing  data  recorded  in  flight  to  improve 
flight  crew  performance,  air  carrier  training  programs  and 
operating  procedures,  airport  maintenance  and  design, 
and  aircraft  operations  and  design”  (Enders,  1993,  p.l). 
UnderTitle  14  of  the  Code  of  Federal  Regulations  (CFR) 
Part  13,  §13.401,  the  FAA  broadens  the  definition  for 
regulatory  purposes  to  include  “routine  collection  and 
analysis  of  digital  flight  data  gathered  during  aircraft 
operations.”  FOQA  programs  evolved  from  accident 
investigation  practices  using  Flight  Data  Recorders 
(FDRs),  which  were  mandated  by  the  Civil  Aeronautics 
Administration  in  1958.  Technological  advances  in  data 
recording,  such  as  Quick  Access  Recorders  (QARs)  and 
Digital  Flight  Data  Recorders  (DFDRs),  in  conjunction 
with  improved  data  management  capabilities,  enabled  the 
analysis  of  routine  data  in  an  effort  to  avert  accidents  and 
incidents  by  identifying  unsafe  practices  or  conditions 
falling  outside  optimal  operating  parameters. 

A  report  published  by  the  U.S .  Government  Account¬ 
ability  Office  (GAO)  in  1997  indicated  that  airlines’  early 
experience  with  FOQA  programs  was  positive.  According 
to  this  report,  FOQA  programs  not  only  enhanced  safety 
but  also  provided  economic  benefits  because  they  were 
“better  able  to  achieve  optimum  fuel  consumption  and 
avoid  unneeded  engine  maintenance.  Although  more  dif¬ 
ficult  to  quantify,  enhanced  safety  should  result  in  lower 
costs  over  time  as  a  result  of  accidents  avoided  and  lower 
insurance  premiums”  (GAO,  1997,  p.2).  Despite  safety 
and  economic  benefits,  as  well  as  endorsements  by  the 
International  Civil  Aviation  Organization  (ICAO),  the 
Federal  Aviation  Administration  (FAA),  the  National 
Transportation  Safety  Board  (NTSB) ,  and  Congress,  vol¬ 
untary  FOQA  participation  has  remained  limited  in  the 
United  States,  particularly  among  small-scale  operators. 

Though  the  majority  of  Part  121  flights  are  operated 
by  airlines  that  have  FAA-approved  FOQA  programs, 
only  1 7%  of  the  smaller  carriers  have  them  (GAO,  2010). 
Reservations  about  FOQA  participation  expressed  by 
some  pilots  and  airline  officials  include  concerns  that 
FOQA  data  might  be  used  for  enforcement,  released  in 
response  to  Freedom  of  Information  Act  (FOIA)  requests, 
or  civil  litigation.  Two  regulations  address  these  concerns: 
The  first  states  that  the  FAA  will  not  use  data  obtained 
from  an  approved  FOQA  program  for  the  purpose  of 


enforcement  except  in  the  case  of  criminal  or  deliber¬ 
ate  acts  (14  CFR  §13.401).  The  second  provides  that 
voluntarily-submitted  information  will  not  be  disclosed 
in  response  to  a  FOIA  request  (14  CFR  §193).  On  the 
other  hand,  civil  litigation  disclosure  remains  a  risk 
because  the  FAA  cannot  restrict  the  authority  of  courts 
through  regulation,  and  legal  precedents  are  mixed  (for 
a  thorough  discussion  of  discovery-related  court  actions, 
see  GAO,  1997,  Appendix  V). 

A  recent  report  cited  pilots’  concerns  about  data  misuse 
as  one  of  the  primary  factors  preventing  participation  in 
voluntary  safety  programs  (GAO,  2010).  These  concerns 
may  extend  to  the  organizations  for  which  they  work, 
even  though  the  threat  of  enforcement  action  or  FOIA 
disclosure  has  been  removed.  Pilot  participation  directly 
affects  voluntary  reporting  programs,  such  as  the  Aviation 
Safety  Action  Program  (ASAP).  The  impact  on  FOQA 
programs  is  more  subtle  and  indirect.  Resentment  and 
concerns  about  data  misuse  may  make  pilots  resistant 
to  training  and  procedural  initiatives  based  on  FOQA 
data.  Considered  in  conjunction  with  pressure  from 
pilot  unions,  airlines  might  find  it  difficult  to  justify  the 
initial  investment  of  implementing  and  maintaining  a 
FOQA  program  if  they  are  dubious  about  its  benefits. 
Thus,  increased  participation  in  FOQA  programs  may 
depend  on  gaining  insight  into  pilots’  perceptions  and 
developing  strategies  to  mitigate  their  concerns. 

The  Perceptions  of  Flight  Operations  Quality  As¬ 
surance  (PFOQA)  questionnaire1  was  designed  to  elicit 
pilots’  level  of  agreement  with  a  series  of  statements  about 
FOQA  programs  using  a  format  widely  recognized  as 
one  of  the  best  for  collecting  information  about  attitudes 
(Nunnally,  1978).  Questionnaire  items  were  based  on 
data  concerns  and  recommendations  proposed  by  the 
Flight  Safety  Foundation  FOQATask  Force  and  reported 
in  the  1 997  GAO  report.  They  were  organized  into  two 
scale  dimensions:  Positive  Perceptions  and  Negative  Per¬ 
ceptions.  As  shown  in  Table  1,  the  Positive  Perceptions 
Scale  comprises  expectations  and  beliefs  about  positive 
safety  enhancements  of  FOQA  programs.  Items  in  the 
Negative  Perceptions  Scale  address  data  misuse  and  orga¬ 
nizational  trust  issues.  The  present  study  is  an  evaluation 


TOQA  was  developed  by  Thomas  R.  Chidester,  Manager  of  the 
FAA  Aerospace  Human  Factors  Research  Division,  and  Thomas  C. 
Accardi,  Director  of  FAA  Aviation  System  Standards. 
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Table  1.  Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA) 
Questionnaire  Items 


Positive  Perceptions  Scale  (9  Items) 

01  FOQA  is  a  program  designed  to  enhance  safety  by  identifying  potential  hazards  before  they 
result  in  an  accident. 

04  Flying  skills  have  improved  or  will  improve  with  a  FOQA  program  in  place. 

06  1  expect  FOQA  data  to  be  used  to  take  action  to  correct  safety  problems. 

07  1  expect  FOQA  data  to  be  used  to  improve  pilot  training. 

08  1  expect  FOQA  data  to  be  used  to  optimize  maintenance. 

10  1  expect  FOQA  data  to  be  used  to  change  cockpit  procedures. 

1 1  1  expect  FOQA  data  to  provide  our  pilot  group  with  useful  feedback  on  our  performance. 

12  1  expect  FOQA  data  to  be  used  to  change  procedures  outside  our  organization  (such  as  in 

Air  Traffic  Control). 

13  1  expect  the  FOQA  program  to  positively  impact  the  safety  of  our  operations. 

Negative  Perceptions  Scale  (7  Items) 

02  (Reflected)  Gatekeepers  are  the  only  persons  able  to  access  identifying  information  that 
associates  a  pilot  or  pilots  with  exceedances. 

03  (Reflected)  1  trust  management  will  not  misuse  FOQA  data  against  individual  pilots. 

05  1  worry  that  FOQA  data  will  be  a  source  of  information  for  enforcement  action  against  pilots. 

09  1  worry  that  FOQA  data  will  be  used  for  disciplinary  actions. 

14  A  FOQA  program  has  negatively  impacted,  or  will  negatively  impact,  the  morale  of  our  pilots. 

15  1  worry  that  FOQA  data  could  be  released  under  the  Freedom  of  Information  Act. 

16  1  worry  that  FOQA  data  could  be  released  through  civil  litigation. 

NOTE:  In  the  computation  of  the  Negative  Perceptions  Scale,  the  two  reflected  items  (02  and  03)  are  scored  as  if  they  were  written 
in  reverse  (i.e.,  “Gatekeepers  are  [NOT]  the  only  persons  able  to  access  identifying  information..."  and  “I  [DON’T]  trust 
management...”). 


of  the  reliability  and  validity  of  the  PFOQA  scales  as  a 
measure  of  pilots’  perceptions  of  FOQA  programs.  Sev¬ 
eral  procedures  will  be  employed  in  this  effort:  Internal 
Consistency  Reliability  Analysis,  Principal  Components 
Analysis,  and  Content  Analysis. 

Internal  Consistency  Reliability  Analysis  determines 
the  extent  to  which  scale  items  appear  to  measure  the 
same  attitudinal  dimension.  Theoretically,  participants’ 
responses  will  be  consistent  within  scales  when  test  items 
represent  a  random  sample  of  all  possible  items  within  a 
particular  content  domain  (Cronbach,  1951).  Responses 
to  poorly  written  items  should  fail  to  demonstrate  internal 
consistency  due  to  random  error  (e.g.,  misread  or  mis¬ 


interpreted  items).  Thus,  internal  consistency  reliability 
analysis  evaluates  both  content  homogeneity  and  item 
quality  (Crocker  &  Algina,  1986). 

Meaningful  internal  structure  is  critical  to  establish¬ 
ing  construct  validity,  so  the  PFOQA  test  items  will  be 
submitted  to  Principal  Components  Analysis  to  examine 
the  underlying  dimensions  in  the  data. The  PFOQAscales 
were  based  on  the  assumption  that  negative  and  positive 
perceptions  of  FOQA  programs  represent  two  distinct 
dimensions  consisting  of  expectations  about  positive 
safety  enhancements  and  concerns  about  data  misuse. 
Principal  Components  Analysis  will  be  employed  to  test 
the  validity  of  this  hypothesis. 
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Content  Analysis  of  participants’  comments  will  be 
used  to  evaluate  the  content  validity  of  the  PFOQA 
questionnaire.  Though  the  questionnaire  comprises  issues 
that  were  deemed  essential  by  experts  in  the  field  (e.g., 
representatives  from  the  FAA,  the  FSF,  and  pilot  unions), 
source  credibility  does  not  guarantee  content  validity. 
It  is  important  to  verify  that  sampling  adequacy  of  the 
questionnaire  items  is  sufficient  to  justify  using  partici¬ 
pants’  scores  to  draw  inferences  about  pilots’  perceptions 
of  FOQA  programs.  Fortunately,  survey  participants  also 
constitute  a  set  of  subject  matter  experts.  Though  the 
volunteer  participants  were  not  asked  directly  whether 
the  PFOQA  items  sufficiently  described  the  underlying 
constructs,  their  responses  to  an  open-ended  question 
located  at  the  end  of  the  survey  (Please  tell  us  anything 
else  you  think  we  should  know  about  your  expectations 
or  concerns  about  FOQA)  should  reveal  issues  that  were 
not  identified  by  the  original  FSF  special  working  group. 

METHOD 

Participants 

Participants  were  199  flight  operations  employees  of 
an  anonymous  commercial  air  carrier.  On-line  data  col¬ 
lection  was  conducted  from  2/9/2010  to  3/30/2010.  Of 
the  1 95  respondents  who  provided  crew  position  infor¬ 
mation,  70  were  first  officers,  109  were  captains,  and  16 
were  check  airmen.  For  evaluating  scale  construction  and 
reliability,  the  sample  was  split  using  a  random  selection 
tool  available  from  SPSS  1 8.0  statistical  software  package. 
Approximately  half  the  cases  were  used  for  the  Principal 
Components  Analysis  (N=  100),  and  the  remaining  cases 
were  used  for  the  Reliability  Analysis  (N=99).  Of  the  1 99 
pilots  who  completed  the  PFOQA  survey,  67  chose  to 
provide  additional  written  comments  that  were  used  for 
the  Content  Analysis. 

Procedure 

The  invitation  to  participate  in  the  PFOQA  survey 
(Appendix  A)  and  a  link  to  the  on-line  survey  (Appendix 
B)  were  embedded  in  the  airline’s  Flight  Operations  web 
site  for  ease  of  access.  This  web  site  was  only  available  to 
the  airline’s  Flight  Operations  personnel,  thereby  pre¬ 
venting  individuals  outside  the  target  population  from 
responding.  Unfortunately,  easy  access  and  participant 
confidentiality  precluded  taking  measures  to  prevent 
participants  from  responding  more  than  once. 


RESULTS 

Descriptive  Statistics 

Summary  descriptive  statistics  for  the  PFOQA  survey 
items  are  shown  in  Table  2.  Frequencies  and  proportions 
for  individual  PFOQA  items  are  provided  in  Appendix 
C.  Though  Likert-type  scales  perform  reasonably  well  in 
parametric  analyses  when  there  are  five  or  more  catego¬ 
ries  (Johnson  &  Creech,  1983;  Zumbo  &  Zimmerman, 
1993),  it  is  unreasonable  to  assume  a  standard  normal 
distribution  based  on  a  4-point  scale  (e.g.,  Berry,  1993). 
Not  surprisingly,  many  of  the  PFOQA  items  deviated 
from  normality,  some  by  more  than  three  standard  devia¬ 
tions.  Item  0 1  (FOQA  is  a  program  designed  to  enhance 
safety  by  identifying  potential  hazards  before  they  result 
in  an  accident),  Item  07  (I  expect  FOQA  data  to  be  used 
to  improve  pilot  training),  Item  13  (I  expect  the  FOQA 
program  to  positively  impact  the  safety  of  our  opera¬ 
tions)  and  Item  16  (I  worry  that  FOQA  data  could  be 
released  through  civil  litigation)  were  particularly  nega¬ 
tively  skewed.  Iteml4  (A  FOQA  program  has  negatively 
impacted,  or  will  negatively  impact,  the  morale  of  our 
pilots)  was  positively  skewed.  Only  Item  07  (I  expect 
FOQA  data  to  be  used  to  improve  pilot  training),  Item 
10  (I  expect  FOQA  data  to  be  used  to  change  cockpit 
procedures),  and  Item  13  (I  expect  the  FOQA  program 
to  positively  impact  the  safety  of  our  operations)  had 
distributions  that  diverged  from  normality  by  more  than 
three  standard  deviations  with  respect  to  kurtosis. 

The  incidence  of  “No  Opinion”  responses  would  result 
in  the  loss  of  more  than  half  of  the  cases  using  pair-wise 
or  list-wise  deletion.  Spearman’s  correlation  between  the 
number  of  user-missing  responses  and  average  response 
items  ( r  =  -.64, p  <.01)  suggests  that  “No  Opinion”  may 
have  been  selected  in  some  instances  when  the  respondent 
intended  to  indicate  strong  disagreement  with  a  question¬ 
naire  item.  Therefore,  item  means  were  substituted  for 
user-missing  values. 

Internal  Consistency  Reliability 

The  results  of  the  Internal  Consistency  Reliability 
Analysis  are  shown  in  Table  3.  By  convention,  a  mini¬ 
mum  Cronbach’s  alpha  of  .80  is  required  for  a  “good” 
scale  (Nunnally,  1978).  Both  the  Positive  Perceptions 
Scale  (a=.86)  and  the  Negative  Perceptions  Scale  (oc=.88) 
demonstrated  good  internal  consistency. 

In  the  aggregate,  indicants  recommended  retention  for 
all  items  in  both  scales.  Values  of  R2,  shown  in  Table  3, 
represent  squared  multiple  correlations  from  regression 
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Table  2.  Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA) 
Questionnaire  Items:  Descriptive  Statistics 


PFOQA  Item 

n 

No 

Opinion 

System 

Missing 

Mean 

SD 

Skewness 

Kurtosis 

01  FOQA  is  a  program  designed  to  enhance 
safety  by  identifying  potential  hazards.... 

197 

2 

0 

3.52 

.58 

-.88 

.69 

02  Gatekeepers  are  the  only  persons  able  to 
access  identifying  information  that.... 

181 

17 

1 

2.94 

.92 

-.66 

-.30 

03  1  trust  management  will  not  misuse  FOQA 
data  against  individual  pilots. 

187 

12 

0 

1.87 

.84 

.41 

-1.03 

04  Flying  skills  have  improved  or  will  improve 
with  a  FOQA  program  in  place. 

184 

14 

1 

3.14 

.72 

-.66 

.60 

05  1  worry  that  FOQA  data  will  be  a  source  of 
information  for  enforcement  action.... 

194 

5 

0 

3.00 

.85 

-.36 

-.77 

06  1  expect  FOQA  data  to  be  used  to  take 
action  to  correct  safety  problems. 

194 

5 

0 

3.32 

.53 

-.10 

.56 

07  1  expect  FOQA  data  to  be  used  to  improve 
pilot  training. 

195 

3 

1 

3.31 

.69 

-1.06 

1.82 

08  1  expect  FOQA  data  to  be  used  to  optimize 
maintenance. 

176 

19 

4 

3.06 

.76 

-.58 

.22 

09  1  worry  that  FOQA  data  will  be  used  for 
disciplinary  actions. 

191 

7 

1 

3.13 

.79 

-.42 

-.74 

10  1  expect  FOQA  data  to  be  used  to  change 
cockpit  procedures. 

186 

11 

2 

3.17 

.59 

-.54 

1.93 

11  1  expect  FOQA  data  to  provide  our  pilot 
group  with  useful  feedback  on  our.... 

191 

4 

4 

3.36 

.62 

-.69 

.91 

12  1  expect  FOQA  data  to  be  used  to  change 
procedures  outside  our  organization.... 

180 

16 

3 

2.93 

.76 

-.41 

-.04 

13  1  expect  the  FOQA  program  to  positively 
impact  the  safety  of  our  operations. 

193 

5 

1 

3.24 

.64 

-.74 

1.58 

14  A  FOQA  program  has  negatively  impacted, 
or  will  negatively  impact,  the  morale.... 

182 

17 

0 

2.03 

.80 

.79 

.60 

15  1  worry  that  FOQA  data  could  be  released 
under  the  Freedom  of  Information  Act. 

179 

20 

0 

3.13 

.82 

-.62 

-.33 

16  1  worry  that  FOQA  data  could  be  released 
through  civil  litigation. 

185 

14 

0 

3.33 

.74 

-.86 

.18 

Note:  Individual  items  were  coded  1  =  Strongly  Disagree,  2  =  Disagree,  3  =  Agree,  4  =  Strongly  Agree 
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Table  3.  Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA): 
Internal  Consistency  Reliability  Analysis  for  Two  Scales  (N  =  99) 


Positive  Perceptions  Scale:  a  =  .86  (9  items) 

R2 

Item-Total  r 

a  If  Deleted 

01  FOQA  is  a  program  designed  to  enhance  safety  by 
identifying  potential  hazards  before  they  result  in.... 

.45 

.60 

.84 

04  Flying  skills  have  improved  or  will  improve  with  a 

FOQA  program  in  place. 

.41 

.52 

.85 

06  1  expect  FOQA  data  to  be  used  to  take  action  to 
correct  safety  problems. 

.44 

.60 

.84 

07  1  expect  FOQA  data  to  be  used  to  improve  pilot 
training. 

.49 

.68 

.83 

08  1  expect  FOQA  data  to  be  used  to  optimize 
maintenance. 

.52 

.65 

.84 

10  1  expect  FOQA  data  to  be  used  to  change  cockpit 
procedures. 

.10 

.17 

.88 

11  1  expect  FOQA  data  to  provide  our  pilot  group  with 
useful  feedback  on  our  performance. 

.62 

.75 

.83 

12  1  expect  FOQA  data  to  be  used  to  change 

procedures  outside  our  organization  (e.g.,  ATC). 

.54 

.62 

.84 

13  1  expect  the  FOQA  program  to  positively  impact  the 
safety  of  our  operations. 

.63 

.72 

.83 

Negative  Perceptions  Scale:  a  =  .88  (7  items) 

R2 

Item-Total  r 

a  If  Deleted 

02  (Reflected)  Gatekeepers  are  the  only  persons  able 
to  access  identifying  information  that  associates.... 

.32 

.51 

.89 

03  (Reflected)  1  trust  management  will  not  misuse 

FOQA  data  against  individual  pilots. 

.54 

.70 

.86 

05  1  worry  that  FOQA  data  will  be  a  source  of 

information  for  enforcement  action  against  pilots. 

.80 

.81 

.84 

09  1  worry  that  FOQA  data  will  be  used  for  disciplinary 
actions. 

.81 

.81 

.84 

14  A  FOQA  program  has  negatively  impacted,  or  will 
negatively  impact,  the  morale  of  our  pilots. 

.27 

.47 

.89 

15  1  worry  that  FOQA  data  could  be  released  under  the 
Freedom  of  Information  Act. 

.64 

.69 

.86 

16  1  worry  that  FOQA  data  could  be  released  through 
civil  litigation. 

.68 

.73 

.86 
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equations  using  each  item  as  the  dependent  variable  with 
all  other  items  as  predictors.  As  such,  larger  values  of  R2 
indicate  that  item  responses  are  well  predicted  by  responses 
to  other  items.  Squared  multiple  correlations  for  Item  02 
{R2=. 32)  and  Item  14  (R2=. 27)  were  slightly  lower  than 
for  most  other  items  in  the  Negative  Perceptions  Scale 
but  not  nearly  as  low  as  Item  10  (f?2=.10)  in  the  Posi¬ 
tive  Perceptions  Scale.  Pearson’s  correlation  coefficients 
of  individual  items  with  total  scores  computed  from  all 
other  items  (Item-Total  r)  demonstrated  comparable 
patterns:  Item  02  (r=.51)  and  Item  14  ( r=A7 )  were 
slightly  lower  than  for  most  other  items  in  the  Negative 
Perceptions  Scale,  but  not  as  low  as  Item  10  (r=.17)  in 
the  Positive  Perceptions  Scale.  In  the  Positive  Perceptions 
Scale,  Item  1 0  had  the  distinction  of  being  the  only  one 
that  would  improve  alpha  by  its  absence  (a  If  Deleted). 
In  the  Negative  Perceptions  Scale,  Items  02  and  14  had 
comparatively  low  Item-Total  correlations  and  R2  values. 
However,  removal  of  these  items  would  produce  a  negli¬ 
gible  increase  in  overall  alpha. 

Principal  Components  Analysis 

Though  the  assumption  of  normality  is  relaxed  when 
PCA  is  used  descriptively,  it  is  sensitive  to  the  magnitudes 
of  correlations  (Tabachnik  &  Fidell,  2006).  The  number 
of  coefficients  with  absolute  values  greater  than  .30  (ap¬ 
proximately  40%)  in  the  Pearson’s  correlation  matrix 
(Table  4)  is  probably  sufficient  for  a  satisfactory  PCA 
solution.  In  addition,  the  Kaiser-Meyer-Olkin  (KMO) 
measure  of  sampling  adequacy  (a  test  of  partial  correla¬ 
tions  among  variables)  for  the  PFOQA  questionnaire 
items  was  .80,  exceeding  the  criterion  of  .60  and  above 
that  is  required  for  a  good  solution. 

PCA  with  Yarimax  rotation  converged  in  three  itera¬ 
tions  and  produced  three  components  with  eigenvalues 
greater  than  1.  As  shown  in  the  rotated  component 
matrix  in  Table  5,  all  variables  had  a  loading  of  .50  or 
greater  with  at  least  one  of  the  components.  The  three 
extracted  components  accounted  for  approximately  60% 
of  the  variance  in  the  dataset.  Component  1  (Negative 
Perceptions)  had  an  eigenvalue  of4.84  and  accounted  for 
24%  of  the  variance.  Items  associated  with  Component 
1  express  concerns  about  data  misuse.  Component  2 
(Positive  Expectations)  had  an  eigenvalue  of  3.64  and 
accounted  for  23%  of  the  variance.  Items  associated 
with  Component  2  involve  positive  expectations  about 
the  benefits  of  FOQA  programs.  Most  items  associated 
with  Component  3  were  positive  declarative  statements 
about  FOQA.  However,  this  component  was  not  as 
clearly  defined  as  the  others.  The  proportion  of  variance 
a  rotated  component  accounts  for  is  an  estimate  of  its 
importance  (Tabachnick  &  Fidell,  2006).  Components 
1  and  2  were  nearly  equivalent  in  this  regard.  On  the 


other  hand,  Component  3  (Positive  Declarations)  had 
an  eigenvalue  of  1 . 1 8  and  accounted  for  only  13%  of  the 
variance  in  the  dataset. 

Content  Analysis 

Comments  were  coded  as  containing  either  positive, 
negative,  or  mixed  (both  positive  and  negative)  content. 
Positive  comments  tended  to  be  broad  and  often  echoed 
statements  included  in  the  PFOQA  questionnaire.  Con¬ 
versely,  negative  remarks  tended  to  be  more  specific  and 
several  expressed  concerns  about  issues  that  were  unrelated 
to  any  of  the  existing  PFOQA  items.  For  example: 

•  Concern  that  FOQA  data  would  not  be  integrated 
with  other  sources  to  address  more  global  problems 

•  Concern  that  exceedance  parameter  definitions  are 
too  narrow 

•  Concern  that  causes  may  be  missed  in  the  focus  on 
event  parameters 

•  Concern  that  a  zero-tolerance  approach  to  exceedances 
will  develop 

•  Concern  for  potential  for  micro-management  of  flying 
through  FOQA 

A  series  of  tests  was  conducted  to  examine  the  charac¬ 
teristics  of  participants  who  provided  comments  relative  to 
the  PFOQA  scales.2  In  general,  pilots  who  provided  com¬ 
ments  had  higher  Negative  Perceptions  Scale  scores  than 
those  who  did  not,  £(197)='3.44,  p<.01.  Of  the  67  (34%) 
pilots  who  chose  to  provide  additional  written  comments, 
11  (16%)  made  only  positive  remarks,  34  (51%)  made 
exclusively  negative  remarks,  and  21  (31%)  made  mixed 
(positive  and  negative)  remarks3  Figure  1  contains  Positive 
and  Negative  Perceptions  Scale  scores  grouped  by  comment 
code.  Participants  who  made  exclusively  positive  comments 
had  significantly  higher  Positive  Perceptions  Scale  scores, 
t(65)=  2.79, p<. 01,  and  significantly  lower  Negative  Percep¬ 
tions  Scale  scores,  t(65)=5.83,p<.01,  than  participants  who 
made  negative  or  mixed  comments.  Participants  making 
exclusively  negative  comments  did  not  differ  significantly 
from  those  with  mixed  comments  on  either  the  Positive 
Perceptions  Scale,  d53)=1.60,p=.12,  or  the  Negative  Percep¬ 
tions  Scale,  d53)=.10,p=.92.This  is  understandable,  given 
the  basic  pattern  of  most  mixed  remarks:  Positive  statements 
were  often  offered  as  recognition  of  the  potential  benefits 
of  FOQA  programs  in  general,  followed  by  negative  com¬ 
ments  detailing  their  concerns  about  their  own  program. 


2As  no  hypothesis  testing  was  involved,  alphas  were  not  adjusted 
for  multiple  analyses.  It  should  also  be  noted  that  Levene’s  tests  for 
equality  of  variances  were  non-significant  for  all  comparisons,  despite 
differences  in  sample  size.  Composite  scale  scores  were  adjusted  for 
the  number  of  items  and  were  normally  distributed. 

3One  comment  was  a  neutral  procedural  recommendation  and  could 
not  be  categorized. 
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'p  <  .01 ;  *p  <.05 


Table  5.  Principal  Components  Analysis  Rotated  Component  Matrix  ( N  =  100) 


Perceptions  of  Flight  Operations  Quality  Assurance 

Component 

(PFOQA)  Questionnaire  Item 

1 

2 

3 

15 

1  worry  that  FOQA  data  could  be  released  under  the  Freedom  of 
Information  Act. 

.87 

16 

1  worry  that  FOQA  data  could  be  released  through  civil  litigation. 

.86 

09 

1  worry  that  FOQA  data  will  be  used  for  disciplinary  actions. 

.85 

05 

1  worry  that  FOQA  data  will  be  a  source  of  information  for 
enforcement  action  against  pilots. 

.78 

03 

1  trust  management  will  not  misuse  FOQA  data  against  individual 
pilots. 

-.72 

07 

1  expect  FOQA  data  to  be  used  to  improve  pilot  training. 

.82 

08 

1  expect  FOQA  data  to  be  used  to  optimize  maintenance. 

.76 

12 

1  expect  FOQA  data  to  be  used  to  change  procedures  outside  our 
organization  (e.g.,  ATC). 

.75 

13 

1  expect  the  FOQA  program  to  positively  impact  the  safety  of  our 
operations. 

.73 

06 

1  expect  FOQA  data  to  be  used  to  take  action  to  correct  safety 
problems. 

.71 

10 

1  expect  FOQA  data  to  be  used  to  change  cockpit  procedures. 

.65 

11 

1  expect  FOQA  data  to  provide  our  pilot  group  with  useful  feedback 
on  our  performance. 

.54 

01 

FOQA  is  a  program  designed  to  enhance  safety  by  identifying 
potential  hazards  before  they  result  in  an  accident. 

.82 

02 

Gatekeepers  are  the  only  persons  able  to  access  identifying 
information  that  associates  a  pilot  or  pilots  with  exceedances. 

.61 

04 

Flying  skills  have  improved  or  will  improve  with  a  FOQA  program  in 
place. 

.58 

14 

A  FOQA  program  has  negatively  impacted,  or  will  negatively  impact, 
the  morale  of  our  pilots. 

-.55 

Component  loadings  <  .50  not  shown. 
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Figure  1.  Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA) 
Questionnaire  Scale  Scores  by  Comment  Code 


DISCUSSION 

Overall,  the  results  suggest  that  the  PFOQA  question¬ 
naire  may  be  sufficient  for  many  applications  in  its  current 
form.  They  also  highlight  aspects  of  the  questionnaire 
that  could  be  improved.  It  is  extremely  important  that 
the  PFOQA  items  represent  a  sufficient  sample  of  pilots’ 
expectations  and  concerns  in  order  to  draw  valid  inferences 
about  their  perceptions  of  FOQA  programs.  The  results 
of  the  Content  Analysis  of  participants’  comments  made 
it  clear  that  the  PFOQA  questionnaire  failed  to  do  this. 
The  FSFTask  Force  (the  source  of  the  PFOQA  question¬ 
naire  items)  concentrated  on  identifying  issues  that  might 
hinder  or  prevent  the  implementation  ofFOQA  programs. 
Our  participants’  comments  included  a  broader  range 
of  concerns  than  those  covered  by  questionnaire  items 
because  they  evolved  from  pilots’  experiences  with  existing 
FOQA  programs  (e.g.,  that  management  will  not  make 
adequate  use  of  the  data,  the  potential  for  zero-tolerance 
policies) .  Consequently,  the  PFOQA  questionnaire  would 
benefit  from  the  inclusion  of  additional  items  based  on 
their  experiences. 

The  results  of  the  Principal  Components  Analysis  also 
suggest  the  need  for  augmentation  of  the  PFOQA  items. 
Only  60%  of  the  variance  in  the  dataset  was  explained 
by  the  extracted  components,  leaving  40%  unexplained. 
Most  was  described  by  the  first  two  components:  Concerns 
about  FOQA  data  misuse  were  associated  with  Compo¬ 
nent  1 ,  and  expectations  ofpotential  FOQA  benefits  were 
associated  with  Component  2.  Ffowever,  Component  3 
was  associated  with  a  few  general  statements  about  FOQA 
programs  that  accounted  for  only  13%  of  the  variance. 


It  is  possible  that  this  component  simply  represents  an 
artifact  of  the  semantic  differences  between  these  items 
and  others,  and  this  component  would  “disappear”  with 
minor  item  revision  (or  analysis  with  a  larger  sample). 

On  the  other  hand,  the  third  component  may  suggest  a 
dimension  that  might  be  developed  using  the  “theoretical 
versus  practical”  duality  evidenced  by  participants’  com¬ 
ments  (i.e.,  positive  statements  about  the  ideal  FOQA 
program,  followed  by  negative  comments  about  their 
own).  Items  that  failed  to  perform  as  well  as  expected  in 
both  the  Principal  Components  and  Reliability  Analysis 
may  need  to  be  rewritten  to  allow  participants  to  express 
this  distinction.  For  example,  Item  02  is  a  general  state¬ 
ment  about  the  role  of  the  Gatekeeper  (Gatekeepers  are 
the  only  persons  able  to  access  identifying  information  that 
associates  a  pilot  or  pilots  with  exceedances) .  Participants 
might  agree  with  the  concept  of  the  Gatekeeper’s  role  as 
stated  and  still  harbor  serious  concerns  about  threats  to 
that  role.  Items  that  specifically  address  these  concerns 
(e.g.,  “I  worry  that  the  Gatekeeper  might  succumb  to 
pressure  from  management  to  release  identifying  infor¬ 
mation”  or  “I  worry  about  management  circumventing 
the  Gatekeeper  and  re-identifying  the  data”)  should 
ameliorate  the  questionnaire’s  ability  to  fully  describe 
pilots’  perceptions  ofFOQA. 

It  is  difficult  to  explain  the  poor  performance  of  Item 
10  (I  expect  FOQA  data  to  be  used  to  change  cockpit 
procedures)  in  the  Reliability  Analysis.  Kurtosis  for  this 
item  diverged  from  normality  by  more  than  three  stan¬ 
dard  deviations  but  cannot  explain  its  low  inter-item 
correlations  and  low  squared  multiple  correlations.  After 
all,  Item  07  and  Item  13  had  similarly  leptokurtotic 
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distributions  but  still  managed  to  produce  excellent  inter¬ 
item  correlations  and  squared  multiple  correlations.  It  is 
possible  that  the  problems  noted  with  Item  1 0  were  due 
to  word  choice.  Changes  to  cockpit  procedures  may  be 
perceived  as  being  positive  or  negative,  and  this  ambigu¬ 
ity  may  have  resulted  in  low  inter-item  correlations  and 
low  squared  multiple  correlations.  Definite  directionality 
(i.e.,  “I  expect  FOQA  data  to  be  used  to  improve  cockpit 
procedures”)  might  improve  its  performance  within  the 
Positive  Perceptions  Scale. 

In  the  aggregate,  the  current  version  of  the  PFOQA 
questionnaire  seems  best  suited  for  assessing  pilots’  at¬ 
titudes  prior  to  FOQA  implementation.  Completion  by 
a  representative  sample  would  offer  a  clear  indication  of 
pilots’  anticipated  benefits  as  well  as  their  concerns  about 
the  program,  and  the  opportunity  to  provide  unstructured 
comments  seems  to  be  extremely  useful  for  identifying 
concerns  that  might  be  unique  to  the  group.  Providing 
feedback  on  questionnaire  results  and  communicating 
how  the  airline  plans  to  address  pilots’  concerns  can  set 
the  stage  for  a  positive  program  introduction.  Certainly, 
the  potential  safety  benefits  of  FOQA  programs  justify 
efforts  to  understand  and  mitigate  negative  perceptions 
that  might  prevent  their  adoption. 
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APPENDIX  A 


Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA)  Questionnaire: 

Invitation  to  Participate 

The  FAA  Civil  Aerospace  Medical  Institute  (CAMI)  in  Oklahoma  City  has  developed  an  online  questionnaire 
to  assess  how  FOQA  programs  are  perceived,  and  has  agreed  to  collect  and  analyze  anonymous  feedback  from 
our  pilots. 

[Airline]  will  benefit  by  receiving  valuable  input  that  might  facilitate  or  improve  implementation  of  our  FOQA 
program.  CAMI  will  benefit  by  having  a  relatively  large  group  of  pilots  take  the  survey  to  establish  its  reliability 
and  validity  as  a  measurement  for  future  participants. 

The  Perceptions  of  Flight  Operations  Quality  Assurance  online  survey  takes  just  a  few  minutes  to  complete. 
Participation  is  completely  anonymous,  which  means  that  no  one  (not  even  the  research  team)  will  know  the 
name  of  any  pilot  who  responds. 

CAMI’s  Aerospace  Human  Factors  Research  Division  conducts  research  in  support  of  the  FAA  Aviation  Safety 
and  Air  Traffic  organizations.  Its  research  is  compliant  with  45  CFR  Part  46  “Protection  of  Human  Subjects” 
and  FAA  Order  9500.25  “Protection  of  Human  Research  Subjects,”  and  is  conducted  under  approval  of  the 
FAA  Institutional  Review  Board.  These  regulations  protect  the  confidentiality  of  participants.  In  this  study,  your 
feedback  is  both  confidential  and  anonymous.  You  will  not  be  asked  for  your  name,  and  personal  identifiers, 
such  as  IP  addresses,  will  not  be  recorded. 

Although  there  is  no  direct  compensation  for  participating  in  this  study,  understanding  your  expectations,  ex¬ 
periences,  and  concerns  is  important  for  making  FOQA  programs  as  effective  as  possible.  Your  participation 
constitutes  a  valuable  contribution  to  our  airline,  the  aviation  community,  and  the  flying  public. 

All  data  that  are  collected  will  be  shared  with  ALPA’s  FOQA  team  here  at  [airline]. 

If  you  consent  to  participate,  simply  login  to  the  Flight  Ops  website  and  click  on  the  button  labeled  “FOQA 
Survey.” 

This  will  take  you  directly  to  the  Perceptions  of  Flight  Operations  Quality  Assurance  questionnaire  which  is 
restricted  to  [airline]  pilots  only. 

Please  remember  that  participation  is  voluntary,  so  you  need  only  respond  to  questions  you  feel  comfortable 
answering,  and  are  free  to  withdraw  from  participation  at  any  time. 

If  you  have  any  questions  about  the  survey,  you  may  contact: 

Tom  Chidester,  Manager 

Aerospace  Human  Factors  Research  Division  (AAM-500) 

Federal  Aviation  Administration 
Civil  Aerospace  Medical  Institute 
6500  MacArthur  Blvd. 

Oklahoma  City,  OK  73125 
(405)  954-4082 
Thomas.Chidester@FAA.gov 


At 


APPENDIX  B 

Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA)  On-line  Survey * 


Please  respond  to  each  item  by  indicating  your  level  of  agreement  with  the  statement 


1. 

FOQA  is  a  program  designed  to  enhance  safety  by  identifying 
potential  hazards  before  they  result  in  an  accident 

Strong y  Agree 

Ayre. 

Doegre* 

ST/oryV  OiMyree 

No  Opnon 

o 

o 

o 

o 

o 

2. 

"Gatekeepers"  are  the  only  persons  able  to  access  identifying 
information  that  associates  a  pilot  or  pilots  with  exceedences. 

Strongy  Agme 

*9— 

Strong!/  Disagree 

No  Opmon 

O 

o 

o 

o 

o 

3. 

1  trust  management  will  not  misuse  FOQA  data  against 
Individual  pilots. 

SrwyV  Agrev 

Agnut 

Drwgree 

Srwyy 

No  Opnon 

o 

O 

O 

o 

4. 

Flying  skills  have  Improved  or  will  improve  with  a  FOQA 
program  in  place. 

Streogy  Agree 

Ayr ee 

ttoeg  re# 

Strong y  Disagree 

No  Opnon 

o 

o 

o 

O 

o 

5. 

1  worry  that  FOQA  data  will  be  a  source  of  information  for 
enforcement  action  against  pilots. 

StrongyAgre* 

Ayr*# 

DiMgree 

Strong^  Dwegree 

No  Opnon 

° 

o 

O 

O 

° 

6. 

1  expect  FOQA  data  to  be  used  to  take  action  to  correct 
safety  problems. 

Strongly  Agm 

Ajn** 

Strong^  Drmgnr* 

No  Opnon 

o 

o 

o 

o 

O 

7. 

1  expect  FOQA  data  to  be  used  to  Improve  pilot  training. 

Strong!/  A.jree 

*9— 

rusagn* 

Strongly  Di*»qi** 

No  Opnon 

o 

o 

o 

o 

o 

8. 

1  expect  FOQA  to  be  used  to  optimize  maintenance. 

ST/ongy  Agree 

Agio* 

D«u*gree 

Strong^  Disagree 

No  Opinion 

O 

O 

0 

O 

O 

9. 

1  worry  that  FOQA  data  will  be  used  for  disciplinary  actions. 

Strongly  Ayr** 

Agio* 

Disagree 

Strong^  Dmyw 

No  Opinion 

o 

o 

o 

o 

O 

10. 

1  expect  FOQA  data  to  be  used  to  change  cockpit  procedures. 

St/ongy  Agree 

Agree 

Du ugree 

Stnvrjl/  Oi soiree 

No  Opinion 

o 

o 

o 

o 

O 

11. 

1  expect  FOQA  data  to  provide  our  pilot  group  with  useful 
feedback  on  our  performance. 

Sl/oryV  Agree 

Agio* 

D'wg*w 

Stwgy  O'wyw 

No  Opnon 

O 

° 

O 

O 

O 

12. 

1  expect  FOQA  data  to  be  used  to  change  procedures  outside 
our  organization  (such  as  in  Air  Traffic). 

Strong!/  Agrt* 

Agio* 

Disagree 

Strong!/  Pis agree 

No  Opnon 

O 

o 

0 

O 

O 

13. 

1  expect  the  FOQA  program  to  positively  impact  the  safety  of 

our  operations. 

STrtxyy  Agree 

Agree 

Disagree 

Stnvgy  Disagree 

No  Opnon 

O 

O 

O 

O 

O 

14. 

A  FOQA  program  has  negatively  impacted,  or  will  negatively 
impact,  the  morale  of  our  pilots. 

Strongy  Agree 

o 

O 

Disagree 

o 

Sfrengy  Disagree 

o 

No  Opinion 

O 

15. 

1  worry  that  FOQA  data  could  be  released  under  the  Freedom 
of  Information  Act. 

Strong*  Agree 

O 

Agree 

O 

Disagree 

O 

Strong!/  Disagree 

O 

No  Opnion 

O 

16. 

1  worry  that  FOQA  data  could  be  released  through  civil 
litigation. 

Strongly  Agree 

o 

Agree 

O 

Disagree 

0 

St/mgy  Disagree 

O 

No  Opnon 

O 

Please  answer  a  few  questions  about  your  background  and  experience. 

17. 

What  aircraft  f)eet(s)  are  you  currently  flying? 

(Ptaasa  chick  all  that  apply  ) 

18. 

Have  you  served  as  a  pilot  on  an  aircraft  fleet  equipped  with  FOQA  prior  to  joining  your  current  airline? 

y&s 
r  no 


1 9.  Where  are  you  based? 

20.  What  is  your  crew  position? 

21 .  What  is  the  highest  level  of  education  you  completed? 

|  -Ok*  Mere-  3 

22.  Please  tell  us  anything  else  you  think  we  should  know  about  your  expectations  or  concerns  about  FOQA. 

(Youi  unadrtad  comments  «*il  be  compiled  »mh  others  and  farwarded  lo  the  aitftne ) 


~3 

d 


Thank  you  for  your  participation! 


’Some  demographic  response  items  have  been  removed  to  protect  the  airline's  anonymity. 
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APPENDIX  C 

Perceptions  of  Flight  Operations  Quality  Assurance  (PFOQA)  Questionnaire  Items: 

Frequencies  and  Percentages 

1  FOQA  is  a  program  designed  to  enhance  safety  by  identifying  potential  hazards  before  they  result 
in  an  accident. 


2  Gatekeepers  are  the  only  persons  able  to  access  identifying  information  that  associates  a  pilot  or 
pilots  with  exceedences. 


Frequency  Valid  Percent 

1 

18 

9.9 

2 

28 

15.5 

3 

82 

45.3 

4 

53 

29.3 

Total 

181 

100.0 

3  I  trust  management  will  not  misuse  FOQA  data  against  individual  pilots. 


Frequency  Valid  Percent 

1 

76 

40.6 

2 

62 

33.2 

3 

46 

24.6 

4 

3 

1.6 

Total 

187 

100.0 

4  Flying  skills  have  improved  or  will  improve  with  a  FOQA  program  in  place. 


Frequency  Valid  Percent 

1 

5 

2.7 

2 

21 

11.4 

3 

101 

54.9 

4 

57 

31.0 

Total 

184 

100.0 

5  I  worry  that  FOQA  data  will  be  a  source  of  information  for  enforcement  action  against  pilots. 


Frequency  Valid  Percent 

1 

7 

3.6 

2 

49 

25.3 

3 

75 

38.7 

4 

63 

32.5 

Total 

194 

100.0 

1  =  Strongly  Disagree,  2  =  Disagree,  3  =  Agree,  4  =  Strongly  Agree 


Cl 


6  I  expect  FOQA  data  to  be  used  to  take  action  to  correct  safety  problems. 


7  I  expect  FOQA  data  to  be  used  to  improve  pilot  training. 


Frequency 

Valid  Percent 

1 

6 

3.1 

2 

8 

4.1 

3 

101 

51.8 

4 

80 

41.0 

Total 

195 

100.0 

8  I  expect  FOQA  data  to  be  used  to  optimize  maintenance. 


9  I  worry  that  FOQA  data  will  be  used  for  disciplinary  actions. 


Frequency  Valid  Percent 

1 

3 

1.6 

2 

40 

20.9 

3 

78 

40.8 

4 

70 

36.6 

Total 

191 

100.0 

10  I  expect  FOQA  data  to  be  used  to  change  cockpit  procedures. 


1  =  Strongly  Disagree,  2  =  Disagree,  3  =  Agree,  4  =  Strongly  Agree 

C2 


11  I  expect  FOQA  data  to  provide  our  pilot  group  with  useful  feedback  on  our  performance. 


Frequency 

Valid  Percent 

1 

2 

1.0 

2 

8 

4.2 

3 

100 

52.4 

4 

81 

42.4 

Total 

191 

100.0 

12  I  expect  FOQA  data  to  be  used  to  change  procedures  outside  our  organization  (e.g.,  ATC). 


Frequency  Valid  Percent 

1  7  3.9 

2  38  21.1 

3  96  53.3 

4  39  21.7 

Total  180  100.0 


13  I  expect  the  FOQA  program  to  positively  impact  the  safety  of  our  operations. 


Frequency  Valid  Percent 

1  4  2.1 

2  10  5.2 

3  115  59.6 

4  64  33.2 

Total  193  100.0 


14  A  FOQA  program  has  negatively  impacted,  or  will  negatively  impact,  the  morale  of  our  pilots. 


Frequency  Valid  Percent 

1 

42 

23.1 

2 

105 

57.7 

3 

22 

12.1 

4 

13 

7.1 

Total 

182 

100.0 

4 


15  I  worry  that  FOQA  data  could  be  released  under  the  Freedom  of  Information  Act. 


_ Frequency  Valid  Percent 

1  6  3.4 

2  32  17.9 

3  73  40.8 

4  _ 68 _ 38^ 

Total  179  100.0 


16  I  worry  that  FOQA  data  could  be  released  through  civil  litigation. 


1  =  Strongly  Disagree,  2  =  Disagree,  3  =  Agree,  4  =  Strongly  Agree 


C3 


